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ABSTRACT 

This paper examined in a critical fashion the 
existing applications of cost-effectiveness analysis in education, 
particularly the study of instructional effectiveness in the 
community college. Various schemes for measuring costs of instruction 
such as cost benefit analysis, cost-effectiveness analysis and 
planning programming budgeting systems were examined and the 
difficulties inherent in using them outlined. Measures of 
effectiveness of instruction were also explored with particular 
attention to measures appropriate for a quantitative analysis. 
Criteria for a useful measure of effectiveness were specified and use 
of measurement of population saturation, achievement, and social 
benefits were examined. Finally, these concepts were applied to a 
variety of real instructional situations, including a comparison of 
the cost-effectiveness of the several methods of teaching remedial 
mathematics employed at Santa Barbara City College (California) 
between September 1961 and September 1971. (Author/AL) 
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X # Introduction 



Administrators and teachers are constantly faced with 
the task of making important, often irreversible, decisions under 
conditions of uncertainty and incomplete knowledge. There is an 
increasing trend in these situations toward the use of rational, 
quantifiable, highly visible, decision-making processes. Systems 
analysis is 

• • • the comparison of alternative means of carrying out 
some function, when tiiose means are rather complicated 
and comprise a number of interrelated elements. Such 
analysis could often be called "economic analysis", 
since the aim is to find the best use of one’s resources 
but the word "systems" is useful in calling attention 
to the complex nature of the alternatives being compared. 

( 25 ) 



Systems analysis thus provides a framework for decision-making and 
its power is that it forces the analyst to make explicit those 
elements of the situation he is taking into consideration. 

The single most important requirement for a systems analysis 
of any system is that there exist a meaningful, unambiguous, and 
quantitative measure of che system output. In large scale educational 
systems the output may be specified in units of students graduated. 
Standardized test scores, increase in lifetime income, or student 
credit hours completed. The major problem in evaluating educational 
systems is the vagueness of specification of objectives and goals 
designed to reflect their output. In addition to more or less specific 
skills (language skills, mathematics, science, etc.) there are subjective 
statements of cultural values, patriotism, aesthetic appreciation, and 
so on. Despite this difficulty it is important that those aspects of 
administrative and instructional planning decisions that are amenable 
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to a quantitative approach be explored and appropriate systems 
analysis techniques developed. , 

It is the function of this paper to examine critically 
existing applications of cost effectiveness analysis in education, 
particularly the study of instructional effectiveness in the 
community college. Various schemes for measuring costs of instruction 
will be examined and the difficulties inherent in this task will be 
outlined. Measures of effectiveness of instruction will be 
explored with particular attention to measures appropriate for 
a quantitative analysis. Criteria for a useful measure of 
effectiveness will be specifics and several possibilities examined. 
Finally, these concepts will be applied to a variety of real instruct- 
lonal situations, including a comparison of the cost-effectiveness 
Of the several methods of teaching remedial ulo themacics, employed at 
Santa Barbara City College between September 1961 and September 1971* 

XX * Cost Effectiveness Analysis in Education 
A. Qvervi ew 

Cost effectiveness implies that we identify costs and 
attendant benefits in some educational system, and that we use a 
combination of the two to evaluate the system. The analysis will 
be used in making decisions concerning resource allocation and the 
selection of alternative strategies in the operation of the system. (17) 
While most teachers are willing, though not eager, to talk about 
instructional effectiveness, they are quite unwilling to consider 



costs* Cost is indelicate, even crass, word in the hallowed ivy halls. 
Nevertheless, we operate today in a climate of scarcity of resouces: 
time, equipment, money, personnel. We do not now have and are unlikely 
to get unlimited resources. Education and individual instructional 
program?; are competing with alternative programs for the same limited 
resources « 

The phrases ,T Cost-Ef f ectiveness Analysis 11 and ,T Cos t~Benaf it 
Analysis 11 are often taken as synonymous. They are not, Alkin (3) 
and ethers (16, 21, 47, 48) have pointed out that one of the main 
requirements of cost-benefit (CB) analysis is that both input and 
output measures be specified in the same units, namely dollars, 

CB analysis is an economic tool used in the examination of both 

public and private sectors of the economy; government* defense, 

business, education. Its chief use is in the examination of historical 
data in an effort to provide prescriptive decision statements , (47) 
Typically CB analysis is the second stage of a systems analysis and 
is most often performed at the level of large scale educational 
systems; state, national, district, or total college level, Bowles 
study of educational planning on the national level (5), Levin’s 
analysis of public education systems (30), Abt’s considerations of 
the voucher system (1,2) and the Hirsch and Marcus study of Junior 
College education (21), are all large-scale CB studies. Many such 
studies seek to determine the social gains, income, and other economic 
benefits resulting from formal education. Planning Program Budgeting 
Systems (PPBS) represents cost-benefit analysis on a big scale, a 
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kind of fiscal systems analysis. (6,33) 



In contrast, cost-effectiveness (CE) analysis may be made 
at all levels, and are less concerned with large-scale economic 
consequences of educational structures as with the evaluation of 
system performance with respect to established objectives and in the 
light of cost factors. CE is primarily a tool for comparison of the 
outcomes of alternative educational decisions. 

A m^jor requirement for the successful use of cost-effectiveness 
analysis is that either the cost or the effectiveness variables be 
limited. According to the common mathematical principles underlying 
maximization, both factors may not vary simultaneoxisly. One must be 
held constant while the other is varied if we are to locate optimal 
values. We may specify the level of instructional efficiency in 
terms of achievement of specified objectives and examine the cost of 
alternative means to reach that level or we may specify an instructional 
budget and compare the effectiveness of alternative instructional 
strategies. (3,24) The equal cost alternative study of Carpenter 
and Haggard (13) is a classic example of the latter option. The most 
serious error made by educational planners applying these systems 
analysis tools is the attempt to locate optimal alternatives by 
allowing both cost and effectiveness to vary simultaneously. Cost- 
Effectiveness means that we look at both system performance and system 
cost when one of these are fixed. (25) 

The typical sequence of actions in a CE analysis iss 
1) Define objectives 
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2) Find a quantifiable output measure for each objective 

3) Set up a cost accounting for the system, hopefully for each 
objective 

4) Construct alternatives. (In a CE analysis of instruction we might 
look at large vs small classes, traditional versus non-traditional 
use of teachers, programmed instruction, tutors, educational 
television. Computer Assisted Instruction, Audio-Tutorial systems, 
etc.) 

5) Test each altemativej either under conditions of fixed cost or 
fixed objectives 

6) Compare relative effectiveness of each alternative (output) 
with cost (input) 

A typical danger in 4) is to assume that a major purpose' is to produce 
** schoo ling.” 

Cost-Effectiveness measures of instructional programs usually 
ignore student inputs, those inputs they bring into the program from 
outside (3) - skills, values, intelligence, personality traits, family 
background, etc. The true effectiveness of instructional sys ter.', is 
given by measures of change in student achievement of objectives. 
Typically this change is measured by course outcomes, test outcomes, 
grades, etc. but should ideally include attitude changes, value 
changes, impact on the community, the teacher, and the school. Long- 
term effectiveness is often implied in system objectives but seldom 
measured. There are very few CE studies of specific instructional 
programs, classroom level programs where the alternatives are ways 
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of instructing, changes in the obviously manipulable variables of 
class size, instructional materials and media, course organization 
or sequence, use of para pro fessional help, and so on, 

B, Cost Measures 

Almost every school budget has a category labeled instruction, 
but the numbers it contains do not represent the cost of instruction. 
Costs of administrative support, library and AV staff, clerks and 
secretarial help, maintenance, classroom construction, teacher 
fringe benefits and many other real costs are not included. Valid 
co'st data is not easily obtained. The costs needed are usually 
combined with other costs., short and long terra costs are not spelled 
out, capital and operating costs may be combined or hidden. 

Further, although the usual measure of cost is money, many 
other costs are expressed in terras of information, time, human effort, 
or lost opportunity, and these may be more important than the dollar 
cost differences in alternatives, Johnson and Dietrich (23) list the 
following cost problems in CE analysis: 

1) Costs are seldom linear . Some costs are fixed, others are 
proportional to the number of students enrolled, and still others 
have a more complex relationship to enrollment, 

2) Cost records are inadequa te , Records may not include sufficient 
detail or may not include historical data, 

3) Not all costs appear in dollar accounts . Opportunity costs do not 
appear. The time of students is considered a free resource, 
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Alternative uses of classrooms or other resources are not considered* 
4) Teacher load data is not reliable * Many uses of teacher time do 
not appear on formal records* 

Despite these problems, the determination of costs for instructional 
programs is far simpler than the determination of effectiveness. 

C. Effectiveness Measures 

It is no surprise, given the diversity, complexity, and 
subtlety of educational endeavors, that effectiveness is difficult 
to measure with validity and reliability* Because education is an 
open system, effectiveness (achievement of objectives) is a more 
valid measure of its optimal functioning than efficiency (relative 
effort required to achieve objectives). (30) However, typical 
educational programs are designed in terms of courses, semesters, 
quarters, fixed time intervals, group achievement rates, and other 
artificial devices that reward efficiency rather than effectiveness. 

This unfortunate fact makes CE analysis and effectiveness measurement 
very difficult* 

There is a need, for CE, to specify objectives concretely, 
to "think through programs and their alternatives in output terms." 

(33) PPBS is simply a way of assuring that educational planners 
focus on measurable objectives* (13,18) The usefulness of 
effectiveness measures is, at best proportional to the specificity 
of the objectives of the system* 

Satisfactory measures of effectiveness listed by Miller and 
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Rath (42) include: 

1) Measures of Population Saturation Pre-test/Post-test gains in 
ability to perform a given objective may be measured for a 
population. If 107. of the students in a course have a certain 
level of achievement on the first day of the course, and 80% 
have reached that level on the last day, we may use these 
figures to measure effectiveness in terms of the relative 
increase in population saturation. 

2) Measures of Achievement Standardized tests may be used to compare 
individual or group achievement with that of a normative population. 
If no pretest is used, achievement scores ignore student input to 
the situation. Achievement tests are best used as before and 
after measures. Most useful are measures of attainment of 
specific objectives or sets of objectives. These may include 

both affective and cognitive measures. 

3) Measures of Social Benefits The social benefits of educational 
programs may be estimated or inferred from real-life situations or 
by the use of appropriate projective tests. 

In addition we may consider long term versus short term 
retention of effectiveness under differing instructional strategies. (49) 



Use of Cost-Effectiveness Indices The purpose of a CE 
analysis is usually to effect the most effective allocation of resource? 
Shubik (45) lists eight ways in which goods and services are usually 
allocated in our society: 

1) Open market competition with a price system 

2) Voting 

- 8 - 
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3) Bidding 

A) Bargaining 

5) Allocation by higher authority or fiat 

6) Allocation by fraud or deceit 

7) Allocation by custom or tradition 

8) Allocation by chance 

Educational systems use 7) almost exclusively and 8) very often* 

Random allocation is a major means of distribution of diseconomies* 

In almost all of these allocation procedures, the effects of persuasion 
and other political activities are an important factor. CE analysis 
provides a basis around which to base political discussions* (44) 

There is a need to wed objective decision-making inputs more effect- 
ively to the political realities of the institution in which the 
decisions are made* 



is the utility of the Ith outcome, C(I) is the cost of that outcome, 
and N is the number of outcomes considered. 



a CE index number, but to acquire the ability to rank alternatives in 
order to obtain a basis for choosing among them. The purpose is 



Traditionally, cost-effectiveness analysis may be summarized 
iji terms of a single ratio or index* Miller and Rath (42) define the 



CE index or CE ratio as 




where P(l) is the probability of occurence of the Ith outcome, U(I) 



The purpose of the CE analysis is not, however, to arrive at 
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formative rather than summative evaluation, in Striven* s terminology. (51) 

Cost-effectiveness analysis is an orderly method of helping decision 

makers to identify a preferred course of action from a set of alternatives. 

The purposes ares 

1) to discover new alternatives 

2) to improve on the existing obvious alternatives (44) 

3) to make more visible non-quantif iable considerations 

4) to provide a means of considering incremental costs (how should 
an additional or reduced amount of money be spent?) 

5) to provide a rational alternative to the use of pure intuition, 
expert opinion, or committee decision in chosing instructional 

strategies. 

Of necessity, any cost-effectiveness analysis is incomplete 
because of intangibles that cannot be included, limitations imposed 
by lack of time, money, information, or uncontrollable changes in the 
system itself. Because of this it is especially important that modem 
operations research techniques - mathematical programming, Leontief I/O 
analysis, Markov process models, linear programming, Monte Carlo 
Simulation, Net-Shift Analysis, and other powerful tools - be used 
with the cost-effectiveness study to extend its usefulness. 

XII. Cost Effectiveness of Various Methods of Instruction; Applications 

The purpose of this portion of the study is to 
1) examine and compare several measures of effectiveness of instruction 
for several courses, and to compare conventional measures based on 
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grades with measures based on behavioral objectives* 

2) determine the relative effectiveness of various methods of 
instruction in a junior college course in remedial arithmetic. 

Xn particular it will compare a system of individualized instruction 
using programmed materials and peer tutors with small traditional 
classes, large classes and independent study. 

3) calculate cost-effectiveness ratios for these methods of instruction 

4) compare four methods of individualized instruction using a 
programmed text and varying numbers of peer tutors. 

5) Examine some of the hidden costs that are important in such a 
cost-effectiveness study, but which are usually ignored. 

6) Consider further possibilities for research into the cost-effectiveness 
of classroom instructional methods. 

A. Remedial Mathematics 

A distinguishing feature of the community college is its open 
door admission policy and the necessity of offering remedial or 
developmental courses in order to assure that marginal students have 
a rasonable chance of success. Basic arithmetic represents a tflajor 
portion of the teaching responsibility of most junior college mathematics 
departments. - The level of success in these courses is depressingly low 
(10), less than 70% in most cases. At Santa Barbara City College where 
the present study was conducted. Math 1, remedial arithmetic serves a 



number of functions: 



1) Xt enables students scoring low on the SCAT pretest to meet the 

math entrance requirements in a variety of courses: elementary 

algebra, physics and other physical science courses, life science 

\ courses, statistics, economics, psychology, pre-nursing, vocational- 
technical courses, business, and so on# 

2) It fulfills the mathematics requirement for the A# A# degree# 

3) It: enables students, who elect to do so, to review their arithmetic 
skills# 

The course content is basically the arithmetic of whole numbers, 
decimals, fractions, and percentage# (10) 

Students enrolling exhibit a wide range of age, educational, 
social, and economic background, occupational and personal interests. 

(11) The assumptions we traditionally make about college students are 
not valid for this group# (38) 

1) They are not motivated to participate in traditional academic 
activities, to succeed academically or to try again after failing# 

2) They are grossly deficient in the conceptual and intellectual skills 
needed for successful academic work in traditional settings# 

3) They lack the attitudes and values needed for success in traditional 
academic situations# 

4) Their out-of-class environment cannot be depended upon to reinforce 
and extend any learning stimulated in the classroom# 

- 12 - 
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5) The range of interests and abilities is extremely large 



Traditional methods of instruction have become associated 
with failure for these students. Many develop an expectation of 
failure and exhibit a pattern of repeated failure. (11) The same 
failure-oriented student is to be found in developmental English, 
Introductory Psychology, Introductory Political Science, and similar 
courses. Instructors usually identify them as not profiting from 
instruction. Their chief characteristic to the admissions officer or 
other administrator is their high dropout rate. Because their every 
encounter with education is colored with the expectation and the 
humiliation of failure# no amount of juggling of AV equipment or 
other hardware and no level of lecturer eloquence will entirely 
offset the pattern of failure. For this reason, remedial mathematics 
at Santa Barbara City College (10) is taught on an individualized 
basis using peer tutors and geared toward the achievement of clearly 
identified behavioral objectives. The system is open with respect to 
course, semester and ether time requirements. All work is achievement- 
oriented; F and D grades are not used, students continue to work with 
their tutors until they have achieved a satisfactory level of completion 
of the course objectives. 

Remedial mathematics at Santa Barbara City College has, by 
both accident and design, undergone a series of changes in the past 
ten years. Foresighted planners in the Mathematics Division and 
the administration have allowed the course to become essentially a 
a research tool for developmental math instruction. The following 
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phases may be identified* 



Phase 1 From Spring 1961 to Spring 1966 Math 1 was taught in the 
conventional manner using small classes, average size 26* 
Unfortunately, grading in these lecture classes was not 
based on specific performance objectives or standardized 
tests* 

Phase 2 From Fall 1966 to Spring 1967 Math 1 was taught with the 

traditional lecture approach and with programmed instruction 
using classes of average size 42* Again, grading was not 
based on specific performance objectives or standardized 
tests* Students were required to attend class* 

Phase 3 From Fall 1967 to Spring 1969 Math 1 was taught using a 

single large class (average 225 during the spring semester and 
508 during the fall semester). Students were not required to 
attend and used self-instructional programmed materials* 

Grades were determined by a single final examination. 

Phase 4 From Fall 1969 to the present Math 1 has been taught as 

individualized instruction using programmed self-teaching 
materials and tutors. Grading is based on performance 
objectives and standardized testing* The instructor acts 
as manager of the instructional system, works with tutors, 
develops materials, etc* rather than in lecture or other 
classroom situations* Attendance at tutoring session is 
required* 



Table 1 shows the grade distribution records for this course in 
remedial math over the interval from 1961-1971* Table 2 shows the 
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corresponding grade distribution for Math 7. Elementary Algebra, a 
si mi lar developmental course, and for the Mathematics Division as 
a whole* Graph 1 shows the changes in P(S) and P(W), the probabilities 
of success and withdrawal for Math 1, remedial arithmetic, during 
the interval 1961 to 1971* P(S) is defined as 



p(s) = 



Total number of A,B, and C grades __ 

Total number of students receiving a grade 



P(W) is defined as 



. . Total number of withdrawn, W, grades _ 

P(W-) — Total number of students receiving a grade 

Notice that in Phase 1, where small, class sections were used, 
the drop probabilities are relatively low and success probabilities 
are high. P(S) is approximately equal to the value for the Mathematics 
Division as a whole: P(S) = 0.72, approximately. P(W) is also 

approximately equal to the value found for the- Mathematics 

Division as a wholes P(W) = 0.25. 

In Phase 2, where larger class sections were held, P(S) 
valuss dropped both for remedial math and for the mathematics division. 
The success rate for elementary algebra, Math7, similarly dropped. 
Withdrawal rates for both courses increased during this phase. 

The changes in P(S) and P(W) for Math 1 cannot be entirely attributed 
to instructional methodology changes because of the corresponding 
changes in Math 7, an ^independent course not undergoing obvious 
changes in procedures. 

In phase 3, where credit by examination and optional 
attendance were adopted, success probabilities dropped still lower 
and withdrawal rates climbed. Division— wide values of these indices 
remained relatively stable. 
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In Phase 4 under Individualized instruction and increased 
resource expenditures, success rates climbed and withdrawal rates 
decreased. Both withdrawal and success rates approached those for 
Elementary Algebra, Hath 7, and for the Mathematics Division as a 

whole. 
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In summary: 

Average P(S) Average P(W) 





Math 1 


Math 7 


Math Div 


Math 1 


Math 7 


Math 


Phase 1 

(small lecture) 
classes 


.72 


.35 


.72 


.25 


.20 


.25 


Phase 2 

(large classes. 


.45 


.36 


.60 


.46 


.40 


.36 


lecture+ PI) 














Phase 3 

(very large 


.30 


.46 


.43 


.60 


.35 


.40 


classes, credit 
by exam) 














Phase 4 


.55 


.65 


.50 


.48 


.28' 


.38 



(individualized 
inst,, behavioral 
objectives, PI, 
tutors) 

Because neither cost nor effectiveness were fixed in 
early versions of the course, any useful cost-effectiveness analysis 
to decide among alternatives is not possible. In Phase 4, however, 
the instructional strategy has changed each semester while a constant 
effectiveness level was maintained. That is, performance requirements 
were held constant in terms of behavioral objectives and standardized 
tests. Course changes involved modification of the tutor/ student 

ratio and attendance reuirements, 

1969 1 tutor, attendance not required 

Spring 1970 4 tutors, attendance not required 

Fall 1970 7 tutors, attendance required in math labs where 
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the student/ tutor ratio was 15/1. 

Spring 1971 10 tutors, attendance required in math labs 

where the student/ tutor ratio was 6/1. 

poth pre- and post-testing of performance in terms of course 
objectives w^s carried out. A cost-effectiveness analysis can thus 
be made fqr the instructional alternatives in Phase 4 but this phase 
cannot be compared with the earlier phases of the course. 

B. COSTS 

^11 costs were calculated on the basis of 1971 dollars and 
1971 salary scales. Only direct instructional costs were included; 
instructor Salaries, paraprofessional salaries, costs of specialized 
equipment amortized over its useful life expectancy. This procedure 
Ignores cost of instructional facilities such as large lecture 

halls versus small classrooms, administrative costs, maintenance 
costs, au4 So on* In calculating the cost per student only those 
students received a grade for the course were counted. Students 

dropping <h«ring the first five weeks of the semester were ignored 
for the purposes of the study. More than 90% of these were ,! no-showsi*’ 

'Tahls 3 gives these costs for Math 1, remedial arithmetic, 
and Math 7 9 elementary algebra. 

4 was designed specifically to treat student time as 
a cost to he taken into account rather than as a free resource. In 
every instructional system one of the very large, but generally intang- 
ible, cost®* is the student time spent. This cost is rarely included 
in instructional cost analysis. In Phase 4 of this study, students 
scheduled their own time, took required examinations ad lib , and 



completed the course as early as they wished. The cost savings 
associated with this procedure could be determined by an analysis 

of student judgements of utility. 

During the fall 1969 and spring 1970 semesters a considerable 
cost savings was effected by scheduling tutor sessions in unused rooms 
during the week before classes began after all other courses had 
been assigned rooms. Normally, the' course would be scheduled into 
specified rooms, thus effectively withholding these rooms from 
alternative uses, a positive cost. By scheduling into unused rooms 
a more effective all-college use was made of available space. The 
cost savings to the college was significant. Tf the course, initial 
enrollment 650 by 1971, was rescheduled into small classes using 
traditional instruction methods, the increase in cost for teacher 
salaries and new classroom facilities required would exceed $40,000. 



(j£ EFFECTIVENESS 

Measures of effectiveness are difficult to assign in an open 
course such as we have in Phase 4. Students may return to complete 
the course at any time and some initially enrolled in 1970 are now 
completing course requirements. The 1970 probability of success, 
P(S)j is still changing. Thus a withdrawn grade may mean that, in 
fact, a student has completed 757. of the course objectives and needs 
only to pass successfully the fourth of the four examinations to 
receive a passing grade. 

Most cost-effectiveness studies performed at the classroom 
level examine only cost per student enrolled or per student credit 
hour. Such studies are useful in costing instructional materials 
(20,22,23,41,46) such as Computer-Assisted Instructi&n (27) or 
Educational Television(26, 39, 40) . These costing studies are easy 



to perform, do not really use a measure of effectiveness, and are not; 
cost-effectiveness analyses. They may be used to make trade-offs on the 
use of some audio-visual materials rather than others, but cost- 
effectiveness trade-offs cannot be made, 

A variety of effectiveness measures have been calculated for 
this study. Most are a form of the CE Index (42) defined on page 9 of 
this paper. 

1 ) Probability of Success , P(s) is the fraction of students 
receiving A’, B, or C grades. In terms of the CE Index, the 
variable I may take on three values corresponding to the 
grades A’, B, and C. P(A) is the probability of receiving an A grade. 
P(B) is the probability of receiving a B grade. P(C) is the probability 
of receiving a C grade. We assume that U(A) — U(B) = U(C) = 1, all 
success grades have equal utility. Non-success grades have zero 
utility. 

2) The weighted probability of success , P.^C S) is defined as- 

P (s) = 4 P(A) + 3 P(B) = 2 P(C). This effectiveness 
w 

measure assignes a greater utility to P(A), U(A) = 4, than 
to P(B), U(B) = 3, or to P(C) ; , U(C) = 2. The weighting corresponds to 
the usual weights used in calculating grade point averages, except 
that grades below C are given zero utility, U(D) = U(F) = U(W) = 0 
Graph 3 shows P w (S) changes for Math 1, Math 7, and the 

Mathematics division during the period 1961-1971, 

3} The success G*P j PjXSl. is defined as 

P ( 5 ) = 4 P(A) + 3 P(B) + 2 P(C) - P(D) - P(F) - P(W). 
g 

This effectiveness measure assigns a negative utility to 
non-success gradest U(A) = 4, U(B) = 3, U(C) = 2, U(D) = UF) = U(W) — 



The success G.P.A. is an attempt to' take into account the 
withdrawal rate as a measure of effectiveness. The difficulty is 
that not all students receiving a W grade would assign it the same 
utility. Many have received the arithmetic review they needed and are 
quite satisfied. They might assign a utility of U(W) = 1 or even higher. 
Others enroll but interact with the system not at all and should not be 
included in an evaluation of the instructional methodology. Unfortunately, 
they are not easily identified. Other students may assign a withdrawal 

grade a utility of -2 or even less. 

Graph 4 shows changes in P (S) for remedial arithmetic, 
elementary algebra, and the mathematics division. Graphs 3 and 4 show 
p (S) and P (S) for the period 1961- 1971 and may be used to compare 

W g 

the effects on grade measures of small classes, large classes, credit 
by examination, and tutoring. 

4) G Statistic, G, developed by McGuigan (33, 34), is defined 

as „ actual gain _ post test ~ pre test 

G ~ possible gain — Maximum -" pre test 

It is a measure of behavioral change. The G Statistic 
assumes the existence of equivalent forms of pre- and post- 
tests with equal total possible score for each. McGuigan presents data 
for 37 programmed and 15 non-pro grammed courses, indicating the 
following percentile equivalents for G: 

G Percentile Percentile 

Programmed Non— programmed 

course Course 

50 49 79 

60 73 80 

70 89 99 

During Phase 4 of the present study, a random sample of approximately 
100 students were pre-tested each semester on the objectives of the 
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coarse. These data may be used to calculate values of G for the 
four instructional alternatives of Phase 4. These values are shown in 
Table A and graph . 

5) The fraction of Objectives achieved , %B0, is the most 

straightforward measure of effectiveness that can be used. 

It is simply the fraction of course objectives achieved 
when those objectives are specified in unambiguous performance 
terms. Unfortunately, few courses of instruction are organized around 
specified behavioral objectives. In this study, only phase 4 made use 
of behavioral objectives. 

No attempt has been made to quantify attitude changes for 
use as a measure of effectiveness, although informal inquiry indicates 
that student norale is much higher under the individualized instruction 
method of teaching than under conventional methods. 

Table 3 s umm arizes the cost-ef f ectivenes indices calculated 
according to Miller and Rath(42) , for the effectiveness measures listed 
above. This index is useful in that it enables us to rank alternative 
methods of instruction used in Phase 4. Not surprisingly, CE indices 
are higher for the Phase 4 program than in most earlier phases. The 
highest CE ratio occurred for fall 1969 when no tutors were employed. 
Withdrawal rates were highest, but costs were lowest. In spring 1970 
tutors were introduced and the CE ratio decreased. In succeeding semesters 
the CE ratio increased as methods of using tutors effectively were 
developed. 

A. great many questions remain unanswered in this analysis. 

In particular, the search for an ideal CE index has not been completed. 

O 
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An ideal CE ratio should (a) increase as effectiveness increases, (b) 
decrease as cost increases, (c) decrease as withdrawal rates increase, 
(d) be sensitive to a weighting of A grades with respect to B or C 
grades, and (e) remain relatively insensitive to number of students 
enrolled. The measures used seem to vary strongly with enrollment. 
Notice that from fall 1966 to spring 1969 low spring enrollments 
produced low CE ratios while high fall enrollments produced high CE 
ratios • 



XV.* SUMMARY AND CONCLUSIONS 



Cost-effectiveness analysis can be applied to classroom 
level instruction as a decision making aid in the selection among 
alternative modes of instruction. Its use as an analytical tool is 
possible only where either cost or effectiveness is held constant across 
the alternatives being considered. Analysis is possible where 
effectiveness is measured in terms of performance objectives or other 

clear-cut unambiguous measures® 

Studies with remedial mathematics courses indicate that, 
although costs are lowest when a credit by examination option, with 
use of programmed instruction and no required attendance, is used, 
effectiveness is correspondingly low and the cost-effectiveness ratio 
is comparable to individualized instruction with programmed text and 
tutors. Intangible cost factors such as room use, student time spent, 
and instructor preparation time make the second alternative much more 
attractive. If the tutors used are peers, students in the same college, 
an effectiveness measure giving the increase in tutor benefits should 
be included. 

Examination of the CE history of an individualized instruction 
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approach Co a remedial echoes course Peaces chac when a Mj „ 
cose increase tabes place, such as extensive use of tutors, chare il 

" In “ iaI ^ C ° St - effeC I— followed „ y a possible 

recurn co higher levels as Inscruocloual planners learn Co use Che 
new resource more effectively. 

This preliminary study suggests th«= f n ii - 
, SS the fo Howxng possibilities 

for future research: 

1) Net shift analysis (8, 12, 32) may be used to i 
_ y USed to develop a measure of 

effectiveness. Ne c shlfc or XnCerval shlfc anal y sls Is a procedure. 

.seed on linear prying Cechnl q ues. which Is useful 1 „ facing 

the degree to which a given set nf e- •. 

some f , , C ° UrSe achlevemenc scores meeCs 

i na l goal score distribution TK i • 

i button. The application of nec shlfc analysis 

o cost-effectiveness analysis might proficably be scudied. 

2) The objective of any course of InsCrucclon Is Co Increase Che. pro. 

cy of success in che course. The probability of success can be 
measured as che course progresses and che se,uence of changes or 
probabilicy cransicions may be CreaCed as scaCes in a Marhov process. 

Then probabilities in a Marbov Cransicion matrix mighc be used as 
measures of effecciveness in a cosc-effecciveness analysis. 

3) Hopefully, che inscruccional ouccomes represented by examination 

scores and behavioral objectives achieved while che scudenc is acdvely 
engaged in the course, reflect inio ♦. 

.. „ long-perm gains in performance. Scudies 

should be performed Co cesc chis assumption in the case of ' 

e case of xns tructional 

programs employing tutors. It is enHr fl i, 

irely conce ivable that reliance on 

tutors during the course may result -in * a 

nay result m a decrease in long-term 

retention. 
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4) There is a need to estimate more accurately the utility of certain 
intangible costs usually overlooked in educational cost-effectiveness 
studies: student time, lessened stress of grades under certain 
instructional methods, value of discretionary time, subjective effect 
of using behavioral objectives, and so on. The utility factors U(l) may 
be estimated from student judgements using multiple linear regression 

techniques. 

5) There is a need to study fixed-cost alternatives using performance 
objectives. This might best be done in Math 7, elementary algebra, 
where historical data is available at Santa Barbara City College* 

'Large class versus small class effectiveness should- be studied in an 
attempt to determine optimal class size. Modifications of the traditional 
approach may also be studied: inclusion of study of programmed materials 
in place of a portion of lecture, use of tutors in or out of class, use 
of individual audio cassettes for explaining difficult topics, substitute 
media for a portion of classtime, and so on. Cost-effectiveness 
analysis may be applied to the evaluation of coordinated instructional 

systems (4) , 4 

6) Experience with peer tutor programs indicates that a great deal of 
learning and positive attitude change takes place in the tutor. <36, 37) 

This is a positive benefit to the tutor and represents a variety of course 
effectiveness that has not been studied from the cost-effectiveness 
point of view* 
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TABLE 1 



MATH 1 GRADE DISTRIBUTION 





A<%> 


B(%) 


c(%) 


w(%) 


Total 


Spiring 7 1 


124(27) 


150(32) 


25(5) 


165(36) 


459 


Fall 70 


81(16) 


175(34) 


20(4) 


236(46) 


512 


Spiring 70 


22(10) 


78(34) 


14(6) 


114(50) 


228 


Fall 69 


15(5) 


97(32) 


36(12) 


155(51) 


303 


Spiring 69 


5(2) 


13(5) 


45(17) 


170(64) 


266 


Fall 68 


36(7) 


36(7) 


98(19) 


283(55) 


514 


Spring 68 


3(2) 


8(4) 


44(22) 


113(57) 


197 


Fall 67 


10(1) 


51(10) 


147(29) 


208(41) 


506 


Spring 67 


17(8) 


28(13) 


55(26) 


104(49) 


212 


Fall66 


25(6) 


71(17) 


88(21) 


171(41) 


417 


Spring 66 


9(7) 


25(20) 


47(38) 


38(31) 


123 


Fall 65 


24(9) 


82(31) 


82(30) 


72(27) 


265 


Spring 65 


10(11) 


19(21) 


37(40) 


20(22) 


92 


Fall 64 


19t8) 


72(30) 


91(38) 


36(15) 


239 


Spring 64 


5(8) 


15(23) 


19(29) 


18(28) 


65 


Fall 63 


8(4) 


58(28) 


90(43) 


40(19) 


207 


Spring 63 


8(13) 


16(25) 


18(29) 


11(17) 


64 


Fall 62 


4(10) 


10(28) 


12(31) 


10(28) 


37 


Spring 62 


5(14) 


7(20) 


7(20) 


9(25) 


36 


Fall 61 


2(8) 


5(17) 


4(26) 


10(35) 


29 


Spring 61 


2(7) 


4(15) 


4(15) 


12(44) 


27 
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TABLE 2 





Math 7 


P(S) 


and P(W) 


Spring 71 


P(S) = 


.64 


pCw) 


Fall 70 




.62 




Spring 70 




.67 




Fall 69 




.25 




Spring 69 




.46 




Fall 68 




.33 




Spring 68 




.36 




Fall 67 




.32 




Spring 67 




.36 




Fall 66 




.31 




Spring 66 




.40 




Fall 65 




.36 




Spring 65 




.44 




Fall 64 




.45 




Spring 64 




.38 




Fall 63 




.70 




Spring 63 




.29 




Fall 62 




.15 




Spring 62 




«15 




Fall 61 




.24 




Spring 61 




.41 






O 

O / 



.36 

.38 

.33 

.75 

.46 

.33 

.37 

.52 

.34 

.36 

.31 

.40 

.40 

.37 

.45 

.20 

.50 

.51 

.35 

.56 

.36 
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(1) at $12,00 per year average 

(2) at $2*25 per hour, 15 hours per week, 15 weeks per semester 



TABLE 4 



Spring 1971 
Fall 1970 
Spring 1970 
Fall 1969 
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Effectiveness Measures for Math 1 



PCs) 


P<W) 


P (s) 

w 


p g Cs) 


G 


% Objectives Achieved 


• 64 


.36 


2.14 


1.78 


.721 


76.8 


.54 


.46 


1.74 


1.28 


.656 


68.9 


.50 


.50 


1.54 


1.04 


.583 


62.6 


.49 


.51 


1.40 


.89 


.552 


59.1 




